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determine  its  value.  We  take  seriously  the  pernicious  effects  of  the  so-called  “theory- 
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reports  (e.g.,  via  overnight  shipping)  to  selected  practitioners  and  policy-makers;  and  most 
notably,  sponsoring  this  symposium,  which  we  craft  intentionally  as  an  opportunity  for 
fruitful,  lasting  connections  between  scholars  and  practitioners. 
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academics’]  problem — it’s  ours  [the  practitioners’].  They  can  only  perform  research;  it’s  up 
to  us  to  use  it.”  While  we  certainly  agree  with  this  sentiment,  we  also  recognize  that  any 
research,  however  theoretical,  must  point  to  some  termination  in  action;  academics  have  a 
responsibility  to  make  their  work  intelligible  to  practitioners.  Thus  we  continue  to  seek 
projects  that  both  comport  with  solid  standards  of  scholarship,  and  address  relevant 
acquisition  issues.  These  years  of  experience  have  shown  us  the  difficulty  in  attempting  to 
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Abstract 

Program  managers  (PMs)  are  expected  to  quantifiably  justify  that  their  program  will 
result  in  the  delivery  of  a  system  with  the  required  performance  through 
development.  Traditionally,  the  PM  has  several  technical  management  tools  at  their 
disposal,  including  Technical  Performance  Measures  (TPMs),  modeling  and 
simulation,  etc.,  that  provide  insight  and  predictive  capability  in  system  performance. 
When  the  program  matures  to  a  point  where  actual  test  data  can  be  gathered,  it  is 
compared  against  expected  system  performance.  The  increasing  use  of  the  system 
of  systems  (SoS)  model  for  the  rapid  fielding  of  warfighting  capabilities  poses  new 
systems  engineering  challenges  for  the  DoD.  Due  to  the  complex  nature  of  SoS 
interdependencies,  PMs  are  especially  challenged  when  asked  to  quantifiably  predict 
progress  made  toward  full-capability  SoS  performance  in  an  incremental 
development.  To  support  the  PM  in  making  technical  trades  and  tracking 
performance  progress  for  an  acknowledged  SoS,  the  U.S.  Navy  (PMS  420  and  SSC 
Pacific)  have  been  collaborating  on  the  development  and  verification  of  an  SoS 
Performance  Measure  (SPM)  tool  set.  The  SPM  tool  applies  a  modified  TPM-type 
approach  to  an  SoS  construct.  However,  instead  of  focusing  on  a  single  measurable 
technical  value  that  can  be  monitored  during  development  of  an  individual  system, 
the  SPM  links  the  SoS  Key  Performance  Parameters  (KPPs)  to  individual  component 
capabilities,  their  maturity,  and  their  potential  usage  rates.  The  System  Maturity 
Model  (SMM),  Concept  of  Operations  (CONOPS),  and  usage  rate  variance  analyses 
are  all  considered  in  the  SPM  calculation.  The  SPM  tool  will  be  reviewed  and 
valuable  lessons  learned  to  date  within  the  Mission  Modules  Program  will  be 
discussed. 

The  Challenges  of  System  of  Systems  Management 

The  Department  of  Defense  (DoD)  has  seen  a  growth  in  the  acquisition  of  systems  of 
systems  (SoSs)  over  the  last  few  decades.  This  trend  is  expected  to  continue  as  the  DoD 
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increases  focus  on  capabilities  without  changing  its  system  oriented  acquisition 
methodologies.  While  providing  significant  opportunities  for  extending  mission  capabilities 
through  the  integration  of  existing  and  new  capabilities  into  a  synergistic  SoS,  there  exists 
significant  systems  engineering  challenges  related  to  the  integration  and  management  of 
SoSs.  These  engineering  challenges  are  discussed  in  the  Systems  Engineering  Guide  for 
Systems  of  Systems  (ODUSD[A&T]SSE,  2008).  While  several  types  of  systems  of  systems 
exist,  the  most  challenging  one  from  the  management  viewpoint  is  the  acknowledged  SoS. 
An  acknowledged  SoS  is  defined  as  one  where  a  set  or  arrangement  of  systems  results 
when  independent  and  useful  systems  are  integrated  into  a  larger  system  that  delivers 
unique  capabilities.  These  capabilities  are  generally  expected  to  exceed  the  capability 
achievable  by  the  component  systems  acting  independently.  An  acknowledged  SoS  has 
recognized  objectives,  a  designated  manager,  and  its  constituent  Mission  Systems  (MS) 
retain  their  independent  ownership,  objectives,  funding,  development,  and  sustainment 
approaches.  The  DoD  Systems  of  Systems  Engineering  Guide  acknowledges  that  the 
acknowledged  SoS  program  construct  poses  a  significant  challenge  to  the  SoS  program 
manager  (PM)  in  trying  to  determine,  monitor,  and  predict  the  technical  maturation  status  of 
the  SoS  during  development  and  integration,  and  as  SoSs  become  more  tightly  integrated 
this  issue  becomes  even  more  challenging.  Asynchronous  development  schedules,  product 
obsolescence,  program  cancellations,  and  the  planned  use  of  technical  insertions  to  take 
advantage  of  technology  improvements  further  increase  this  management  challenge  for  the 
PM. 

Within  this  challenging  environment,  the  role  of  the  acknowledged  PM  remains  to 
balance  cost  and  schedule  while  managing  risk  to  provide  a  desired  level  of 
capability/performance.  In  the  area  of  predicting  performance,  the  challenge  presently 
facing  SoS  PMs  is  the  understanding  of  the  SoS’s  technical  level  of  integration  and  maturity, 
and  then  relating  that  data,  especially  during  development,  to  the  predictions  of  performance 
to  be  achieved.  Historically,  the  Technical  Performance  Measure  (TPM)  methodology  has 
been  used  as  a  key  gauge  of  the  probability  of  an  individual  system  meeting  its  performance 
objective  when  development  is  complete.  Unfortunately,  in  an  SoS  context  this 
methodology  often  fails  to  provide  the  desired  insight  for  several  reasons  including  the  fact 
that  TPM  data  at  the  system  level  may  not  be  obtainable  by  the  SoS  PM,  the  metric  at  the 
system  level  may  not  be  relevant  to  how  the  SoS  will  employ  the  existing  system,  and  the 
end  user  may  have  a  variety  of  options  in  employing  the  components  of  the  SoS  to  achieve 
the  desired  end  capability  so  that  a  single  point  value  may  be  of  minimal  use.  Aside  from 
TPMs,  modeling  and  simulation  (M&S)  have  also  been  used  to  provide  insights  into 
performance.  However,  these  methods  are  often  costly  and  time  consuming  to  conduct, 
thus  limiting  their  usefulness  in  the  fast  paced  world  of  acquisition.  Additionally,  this 
approach  suffers  from  two  additional  challenges  in  an  SoS  application.  The  first  issue  is  that 
the  model  or  simulation  developed  for  the  system  level  capability  may  not  be  reflective  of  its 
operational  performance  or  usage  within  the  SoS.  This  could  result  in  the  need  for  costly 
redevelopment  and  validation  of  the  model  or  simulation  and/or  the  development  of  new 
interfacing  software.  The  second  issue  is  that  today’s  SoSs  often  enable  the  use  of  their 
individual  component  systems  in  various  combinations  and  in  various  operational  methods 
to  accomplish  the  top  level  tasking.  Thus  any  single  model  run  may  not  provide  the  overall 
insight  into  the  range  of  performance  the  SoS  may  actually  provide  and  could  lead  to  a 
misunderstanding  of  the  SoS  capabilities. 

This  paper  will  review  the  use  of  metrics  in  acquisition  and  then  provide  a  brief 
overview  of  various  methodologies  and  metrics  presently  used  or  proposed  for  providing 
insight  into  the  management  of  systems  of  systems,  including  those  being  developed  and 
tested  by  the  Littoral  Combat  Ship  (LCS)  Mission  Modules  Program  Office  (PMS  420).  The 
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paper  will  then  seek  to  address  what  appears  to  be  a  gap  within  the  management  and 
oversight  of  SoS  development.  Specifically,  while  various  methods  exist  for  supporting  the 
PM  in  the  challenge  of  monitoring  SoS  development,  an  area  that  does  not  seem  to  have 
been  extensively  addressed  in  the  literature  is  the  challenge  of  predicting  performance  for 
an  SoS.  To  address  this  challenge,  an  approach  being  investigated  by  PMS  420  will  be 
presented.  This  method  involves  the  development  of  a  proposed  metric  called  the  SoS 
Performance  Measure  (SPM)  which  will  be  discussed  and  a  notional  case  study  will  be 
presented. 

Metrics  and  Their  Limitations 

When  discussing  metrics,  perhaps  the  first  issue  to  be  clarified  is  the  definition  of  a 
metric.  Figure  1  shows  the  Merriam-Webster  online  dictionary  definition  of  the  term  metric 
(“Metric,”  n.d.).  For  the  DoD,  the  second  definition,  “a  standard  of  measurement, “appears  to 
be  the  most  relevant  in  terms  of  how  the  metric  is  used  in  acquisition.  Selection  of,  and 
concurrence  with  the  selection  of  the  definition  is  a  critical  issue  in  the  discussion  to  follow  in 
part  because  the  third  definition  of  a  metric  is  “a  mathematical  function  that  associates  a  real 
nonnegative  number  analogous  to  distance  with  each  pair  of  elements  in  a  set  such  that  the 
number  is  zero  only  if  the  two  elements  are  identical,  the  number  is  the  same  regardless  of 
the  order  in  which  the  two  elements  are  taken,  and  the  number  associated  with  one  pair  of 
elements  plus  that  associated  with  one  member  of  the  pair  and  a  third  element  is  equal  to  or 
greater  than  the  number  associated  with  the  other  member  of  the  pair  and  the  third 
element.”  Metrics,  as  generally  used  within  the  DoD,  are  normally  viewed  to  reflect  the  use 
of  ordinal  or  interval  data  vice  ratio  scaled  data.  Whereas  arguments  exist  with  respect  to 
this  classification  and  its  mathematical  implications,  its  primary  impact  is  in  terms  of  usability 
to  a  PM.  In  terms  of  operational  utility  for  the  PM,  what  it  means  is  the  need  to  understand 
that  the  metric  data  presented  for  their  effort  is  best  used  for  providing  informational  insight 
into  the  specific  project  being  analyzed  and  may  not  have  validity  when  compared  as  a 
measure  against  other  efforts.  In  other  words,  we  should  use  metrics  for  understanding 
trends  within  a  program  and  not  count  on  them  for  precision  answers. 

Does  this  limitation  impact  the  utility  of  metrics?  This  paper  argues  no.  In  the  world 
of  defense  acquisition,  insight  is  often  what  is  required,  and  metrics  have  been  used 
successfully  to  provide  that  insight  to  decision-makers.  For  effective  program  execution  what 
an  SoS  or  system  PM  is  often  seeking  is  insight  into  their  immediate  status,  the  ability  to 
predict  whether  they’re  on  track  to  provide  the  requisite  performance  within  cost  and 
schedule  constraints,  and  to  understand  the  impacts  of  the  various  options  they  may  have  to 
choose  between  when  executing  their  program.  Many  of  the  metrics  are  well  known  and  are 
often  expressed  as  readiness  levels  including  the  evaluation  of  Technology  Readiness  Level 
(TRL),  Manufacturing  Readiness  Level  (MRL),  and  Earned  Value  (EV).  Newer  metrics  that 
are  becoming  of  increased  use  within  the  acquisition  community  include  those  of  Software 
Readiness  Levels  (SwRL),  Integration  Readiness  Levels  (IRL),  and  System  Readiness 
Levels  (SRL).  Many  of  these  metrics  operate  by  comparison  of  a  product  against  a  known 
scale  to  determine  a  present  value  (TRL,  MRL)  which  can  then  be  compared  against  the 
program’s  status  and  against  historical  data  to  indicate  if  a  program  is  on  track  against  its 
present  developmental  stage.  Others  such  as  EV  and  SRL,  while  based  on  known  status, 
can  also  be  used  in  a  manner  similar  to  TPMs  to  provide  indications  of  whether  the  program 
is  on  a  trend  to  achieve  desired  objectives  or  if  a  potential  risk  exists.  Let  us  now  look  at  how 
an  executing  program  office  is  using  these  tools  within  an  acknowledged  SoS. 
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The  Mission  Modules  Program  Office — Developing  Tools  for  Understanding, 
Predicting,  and  Managing  an  Acknowledged  SoS 

The  LCS  Mission  Modules  Program  Office  (PMS  420)  was  established  by  the 
Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition  (ASN  RD&A) 
on  October  1 , 2003,  within  the  Program  Executive  Office,  Littoral  and  Mine  Warfare  (PEO 
LMW)  for  the  development,  acquisition,  and  sustainment  of  the  modular  mission  packages 
(MPs).  The  initial  focus  of  PMS  420  was  to  take  existing  independent  capabilities  in  the 
fields  of  surface  warfare  (SUW),  mine  countermeasure  (MCM),  and  anti-submarine  warfare 
(ASW)  and  to  integrate  and  modularize  those  capabilities  to  provide  deployable  and 
swappable  warfighting  capabilities  for  the  LCS.  Thus  the  LCS  MPs  meet  the  definition  of 
being  an  SoS  because  they  are  made  up  of  individual  MSs,  including:  vehicle, 
communication,  sensor,  or  weapon  systems;  support  equipment,  including  support 
containers,  or  vehicle  cradles;  software;  mission  crew  detachments;  and  aviation  systems, 
which  are  then  integrated  into  a  larger  system  to  deliver  unique  capability.  Because  the 
charter  of  PMS  420  is  to  acquire,  integrate,  modularize,  and  sustain  focused  warfighting 
capabilities  from  existing  program  lines,  PMS  420  primarily  serves  as  a  Ships  Acquisition 
Program  Manager  (SHAPM)  with  a  focus  on  acquiring  the  individual  mission  systems  from 
Participating  Acquisition  Resource  Managers  (PARMs),  who  manage  existing  product  lines 
and  programs  of  records.  This  lack  of  direct  management  responsibility  for  the  individual 
mission  systems  means  that  the  SoSs  comprising  the  MPs  are  acknowledged  SoSs.  Since 
its  founding,  PMS  420  has  recognized  the  challenge  of  leading  an  acknowledged  SoS 
development  and  quickly  began  development  of  novel  system  engineering  tools  and 
methodologies,  designed  to  ultimately  reduce  risk  and  provide  enhanced  management 
(technical,  cost,  schedule)  insight  into  the  SoS  problem.  Initial  tools  addressing  some  of  the 
traditional  programmatic  concern  related  to  determining  technology  readiness, 
understanding  technology  insertion  options,  and  managing  investments  have  been 
developed  and  are  being  used  by  PMS  420  on  a  daily  basis.  Lessons  learned  by  PMS  420 
related  to  these  issues,  the  approaches  used,  and  their  benefits  have  been  discussed 
previously  at  this  symposium  (Volkert,  Jackson,  Harper,  &  Van  Norstrand,  2010). 

One  of  the  primary  methods  used  by  PMS  420  to  gain  insight  and  manage  the 
development  maturity  of  their  SoS,  as  presented  to  this  symposium  in  2009,  has  been  the 
concept  known  as  the  SRL  developed  by  Dr.  Sauser  (Forbes,  Volkert,  Gentile,  &  Michaud, 
2009).  By  pairing  the  traditional  TRL  scale  with  a  new  series  of  criteria  known  as  the 
Integration  Readiness  Level  (IRL),  a  more  complete  look  at  true  system  maturity  can  be 
obtained  (Sauser,  Ramirez-Marquez,  Magnaye,  &  Tan,  2008).  Under  this  methodology  the 
readiness  of  each  technology  is  still  considered,  but  instead  of  being  a  stand-alone  metric 
for  determining  readiness  for  incorporation,  it  is  analyzed  in  concert  with  both  its  integration 
requirements  and  the  maturity  of  other  technologies  with  which  it  interfaces.  The  calculation 
of  SRL  is  described  in  the  above  referenced  papers.  The  SRL  methodology  has  been  highly 
successful  on  the  program  and  has  paid  dividends  in  terms  of  both  increasing  decision¬ 
maker  visibility  into  true  system  status  and  allowing  for  pre-emptive  actions  to  be  taken  to 
mitigate  potential  developmental  issues.  Since  the  initial  presentation  of  the  SRL  method, 
the  program  has  developed  and  documented  a  comprehensive  process  for  System  Maturity 
Assessments  (SMA)  and  has  described  its  application  to  generic  SoSs.  The  SMA  process  is 
iterative  with  a  structured  set  of  well  developed  tasks  that  are  described  in  detail  in  the 
System  Maturity  Assessment  Guide  (PMS  420,  2009).  The  first  three  steps  of  this  process 
need  only  be  conducted  during  initial  system  architecture  development.  Once  the  system 
architecture,  a  key  factor  noted  in  many  of  the  methodologies  reviewed  for  defining 
parameters  within  an  SoS,  and  subsequent  system  designs  have  been  placed  under 
configuration  control,  successive  assessment  iterations  need  only  review  the  previous  TRL 
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and  IRL  criteria  for  any  updates  due  to  development  progress  and  then  recalculate  the  SRL 
with  updates  to  reporting  mechanisms  conducted  as  needed.  The  fundamental  basis  of  the 
SMA  process  is  the  proper  creation  of  an  assessment  framework  to  include  technologies, 
integrations,  and  their  resulting  architecture.  It  is  also  imperative  that  buy-in  from  all 
stakeholders  be  obtained  in  order  to  ensure  common  understanding  among  all  participants 
with  regard  to  both  what  will  be  evaluated  and  in  what  manner.  The  SMA  model  has  been 
applied  for  the  purpose  of  monitoring  the  maturity  and  integration  status  of  individual 
technologies  within  the  MP  SoSs  for  PMS  420.  The  Mission  Modules  Program  has  used  this 
methodology  to  monitor  developmental  status  by  incorporating  it  into  a  continuing  quarterly 
evaluation  of  the  SRL  level  for  each  of  the  mission  packages.  This  consistent  evaluation 
allows  the  PMS  420  PM  to  better  understand  maturation  of  the  individual  MP  SoS  and  of 
each  increment  within  the  SoS.  In  turn,  this  provides  him  with  a  greater  understanding  of  the 
program’s  technical  status,  enabling  the  PM  to  better  maintain  and  manage  the 
development  risk  of  the  MPs  as  they  progress  though  design  and  development. 

However,  as  viable  as  the  above  tools  have  been  to  PMS  420  in  managing  its  SoSs, 
the  tools  still  fail  to  provide  insight  into  one  of  the  most  asked  questions  of  the  PM.  Will  the 
SoS  obtain  the  performance  desired  and  within  the  incremental  development  approach, 
when  and  with  what  combination  of  capabilities?  For  this  issue  a  new  approach  is  required. 

Relationship  of  Performance  Prediction  to  Technical  Measurements 

The  International  Council  on  Systems  Engineering  (INCOSE)  has  published  a 
technical  measurement  guide  that  lays  out  the  standard  acquisition  approach  for  relating 
operator  requirements  to  quantifiable  and  measureable  data  that  can  be  obtained  during 
development  (Roedler  &  Jones,  2005).  To  set  the  stage  for  the  proposed  methodology  this 
present  methodology  is  summarized  as  follows.  Traditionally,  within  defense  acquisition 
system  development,  performance  expectations  for  a  developmental  program  are 
established  by  defining  a  set  of  criteria  called  the  Measures  of  Effectiveness  (MOEs).  MOEs 
are  generally  used  within  the  DoD  community  to  define  the  level  of  operational  success 
desired  of  the  developing  capability  that  is  related  to  the  mission  and  environment  for  which 
the  system  is  being  developed.  They  represent  the  end  users’  desires  for  system  capability 
in  terms  of  operational  value  vice  any  specific  technical  approach.  Critical  elements  of  the 
MOE’s  generally  get  translated  into  the  Key  Performance  Parameters  (KPPs)  of  a  system 
that  are  used  to  help  drive  the  critical  performance  aspects  of  a  proposed  design.  As  a 
system  design  concept  evolves  for  providing  the  performance  required  by  the  MOEs  and  the 
KPPs,  Measures  of  Performance  (MOP)  are  then  developed.  The  MOPs  for  a  system 
represent  selected  physical  and  functional  characteristics  related  to  the  systems’  operations 
that  can  be  measured  during  testing/operations  and  used  to  indicate  that  the  performance 
requirements  of  the  MOEs  and  KPPs  are  being  achieved  by  the  design  being  evaluated. 
Because  testing  generally  occurs  late  in  the  program  development  cycle,  an  indicator  of  the 
desired  performance  attainability  prior  to  the  gathering  of  MOP  data  is  desired  to  justify  the 
ongoing  investment.  At  the  single  system  level  this  is  obtained  through  the  selection  of 
TPMs.  TPMs  represent  an  ongoing  measurement  of  a  technical  requirement  where  the 
technical  requirement  being  measured  is  assumed  to  have  a  direct  relationship  to  the 
eventual  accomplishment  of  a  MOP,  see  Figure  1  (DoD,  2003).  Trend  monitoring  of  the 
various  TPMs  of  a  program  towards  their  goals  is  then  used  as  an  indicator  that  the  system 
should  eventually  accomplish  its  required  performance  goals. 
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Figure  1.  Technical  Parameter  Measure  Notional  Chart 

Why  Individual  System  TPM’s  May  Fail  to  Provide  Insight  for  a  System  in  an 
Acknowledged  SoS 

While  TPMs  have  been  used  successfully  at  a  system  level  to  provide  indication  of 
end  product  performance  related  to  customer  needs,  this  methodology  presents  challenges 
within  an  acknowledged  SoS  for  several  reasons.  First,  because  the  SoS  PM  does  not  have 
direct  control  over  the  developmental  PMs  gaining  and  maintaining  status  of  the  individual 
system,  TPMs  may  not  be  achievable.  Second,  and  perhaps  more  relevant,  is  that  even  if 
the  data  is  obtainable,  it  may  not  be  relevant  with  respect  to  the  systems  use  within  the  SoS. 
One  of  the  key  aspects  of  an  SoS  is  that  enhanced  performance  is  achieved  by  the 
integration  of  the  individual,  independent,  and  useful  systems  into  a  larger  system  that 
delivers  unique  capabilities.  These  unique  capabilities  are  often  achieved  through  changes 
to  the  operational  use,  maintenance  support,  operational  environment,  or  employment 
methodology  of  the  individual  system.  For  example,  a  system  originally  developed  for 
airborne  use  where  weight  is  often  a  critical  TPM  may  be  reconfigured  in  an  SoS  for  use  off 
of  an  unmanned  vehicle  (UV).  In  this  application,  the  weight  may  no  longer  be  a  critical 
factor  for  the  SoS,  but  reliability  may  be.  Because  airborne  systems  are  generally  used  on 
missions  of  relatively  short  durations,  what  was  acceptable  reliability  before  for  the  system 
may  not  be  operationally  effective  for  a  long  duration  UV  application.  Thus,  the  carryover 
and  monitoring  of  just  the  system  level  TPMs  from  the  component  systems  could  result  in  an 
erroneous  view  of  the  SoS’s  capabilities.  Additionally,  as  systems  are  incorporated  into  the 
SoS  they  are  often  integrated  with  other  components  of  the  SoS.  This  integration  may  result 
in  some  of  the  initial  capabilities  of  the  individual  system  being  either  enhanced  (such  as 
interfacing  an  improved  sensor  with  an  existing  detection  system)  or  degraded  (removing  a 
data  input  previously  provided  by  another  interfacing  system  that  is  not  included  within  the 
SoS’s  capability  set).  For  these  reasons,  a  methodology  for  predicting  performance  within 
the  sphere  of  control  of  the  SoS  PM  that  can  account  for  the  uniqueness  of  the  SoS 
approach  is  required. 

Criteria  for  a  Useful  SoS  Performance  Measure  (SPM)  Metric 

For  a  metric  to  successfully  assist  an  SoS  PM  in  predicting  performance  we  must 
understand  some  of  the  key  drivers  in  SoS  development  and  fielding.  As  discussed 
previously,  one  key  criterion  is  the  need  for  the  SoS  PM  to  be  able  to  develop  a  level  of 
insight  with  respect  to  the  ability  of  their  SoS  to  obtain  the  desired  performance  levels 
without  detailed  insight  into  the  detailed  technical  capabilities  and  limitations  of  the 
component  systems.  In  addition,  the  methodology  will  need  to  be  able  to  adjust  known 
system  performance  data  for  the  impacts  to  be  expected  when  operating  within  the 
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integrated,  operational,  and  environmental  constraints  of  the  SoS  vice  those  that  the 
individual  system  was  developed  for.  Because  SoSs  often  offer  greater  flexibility  in  the 
assignment  of  tasking  among  the  component  capabilities  than  may  have  existed  at  the 
individual  system  level,  this  range  of  operational  employment  details  and  its  impact  on 
usage  rates  will  need  to  be  considered.  Finally,  because  an  SoS  reflects  a  combination  of 
capabilities,  it  is  reasonable  to  assume  that  as  subsets  of  these  capabilities  mature  the 
incremental  fielding  of  the  capabilities  will  occur.  Also,  the  SoS  will  face  a  greater  need  than 
a  single  system  to  be  able  to  deal  with  the  challenges  posed  by  technology/capabilities  both 
maturing  at  various  rates  than  may  have  been  initially  planned  and  also  those  that  go 
obsolete  and  require  replacement.  A  metric  called  the  SoS  Performance  Measure  (SPM)  is 
an  attempt  by  PMS  420  to  answer  these  challenges. 

SoS  Performance  Measure  (SPM) 

To  arrive  at  a  useable  metric  for  the  SoS  PM  we  must  start  to  define  the  metric  in 
terms  of  the  criteria  discussed  above.  Therefore  we  start  by  defining  the  SoS  Performance 
Measure  as  follows: 

SPM(sos  evaluated)  =  f(  SoS  capability,  operational  employment)  (1 ) 

Where  the  SoS  capability  is  defined  as  a  combination  of  the  impacts  of  the  system 
and  SoS  technical  maturity,  SoS  integration  impacts,  SoS  support  impacts,  and  weighted 
system  performance.  For  the  operational  employment  component  of  SPM  we  consider  the 
planned  usage  options  (can  a  system  in  the  SoS  help  meet  a  performance  goal),  and 
potential  usage  rates  (how  much  will  it  be  used)  under  a  range  of  perspective  Concept  of 
Operations.  Once  we  have  the  broadly  stated  functions,  we  must  then  determine  how  to 
convolve  the  values  and  what  constraints  are  imposed  in  the  selection  of  the  values  being 
used  to  determine  SPM.  Basically,  we  must  answer  the  question  of  can  we  define  the 
component  elements  of  the  function  in  a  meaningful  way  that  provides  value  to  the  PM? 

To  initiate  this  activity  some  preliminary  work  is  required  by  the  SoS  PM  and  their 
staff.  Specifically,  a  well  defined  architecture  for  the  SoS  is  critical.  Without  this  baseline  tool 
to  define  the  systems  composing  the  SoS  and  their  interactions,  analysis  of  performance, 
risk,  or  maturity  is  challenging.  With  respect  to  the  SPM,  the  architecture  and  interaction  of 
the  systems  in  the  SoS  is  needed  to  enable  the  mapping  of  how  the  individual  systems 
contribute  towards  the  performance  metrics  (KPP  for  this  case)  and  to  assist  the  operation 
users  in  developing  the  various  operational  usage  concepts. 

SoS  Capability  Components  (SoS  Technical  Maturity,  SoS  Integration,  SoS 

Support,  &  System  Performance) 

Accounting  for  SoS  Component  Technical  Maturity  &  Integration.  Seeking  not  to 
reinvent  the  wheel,  the  SPM  methodology  using  SRL  data  can  be  used  to  provide  insight 
into  the  status  of  the  SoS  in  terms  of  maturity  and  integration.  While  the  use  of  SRL  is  a 
subject  of  debate  with  respect  to  the  validity  of  its  mathematical  methodologies,  what  it 
provides  is  an  effective  risk  evaluation  of  a  set  of  capabilities  combined  together.  What  the 
application  of  the  SRL  methodology  provides,  constrained  against  the  defined  architecture, 
is  a  defined  and  repeatable  process  for  arriving  at  a  single  point  value  for  representing  the 
SoS.  While  not  a  perfect  answer,  and  as  with  all  metrics  requiring  the  application  of  a 
healthy  amount  of  skepticism  to  what  its  results  really  represent,  it  is  in  effect  no  worse  than 
the  mental  integration  of  capabilities  into  a  single  value  that  we  are  often  called  upon  to  do 
daily  with  complex  systems. 
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Accounting  for  SoS  Support.  As  discussed  previously,  one  of  the  impacting  factors 
in  SoS  performance  is  the  delta  between  the  supportability  concept  that  the  original  system 
was  developed  under  and  the  implications  of  the  support  concept  to  be  used  by  the  SoS. 
These  deltas  can  have  either  a  positive  or  negative  impact  on  the  system  and  on  SoS 
capabilities.  In  the  early  stages  of  SoS  development,  this  delta  may  not  be  fully  quantifiable 
in  terms  of  impact,  but  as  the  SoS  develops  it  should  become  clearer  to  the  PM.  Because 
the  eventual  purpose  of  the  SPM  metric  is  to  provide  insight  and  support  sensitivity  analysis 
and  to  support  the  ease  of  calculations,  the  use  of  a  weighting  factor  will  need  to  be  applied 
to  the  existing  supporting  concept,  which  is  notionally  set  to  1 .0  to  account  for  impact.  Thus, 
the  impact  at  a  system  level  can  be  defined  as: 

Ssos=IcOiSi/ZSi  (2) 

where  i  =  1  ,..n  for  n  systems  in  the  SoS,  and  where  the  notional  level  of  SoS  supportability 
reflects  the  sum  of  the  weighted  supportabilities  of  the  individual  systems  within  the  SoS 
normalized  against  their  existing  level  of  supportability  as  a  standalone  system. 

Accounting  for  System  Performance.  System  performance  varies  over  time 
generally  in  accordance  with  where  the  system  stands  in  terms  of  its  developmental 
maturity.  Figure  2  provides  a  notional  example  of  this  performance  maturation  mapped 
against  a  curve  looking  at  historical  maturation  stages  reflective  of  developmental 
milestones  such  as  Advanced  Developmental  Models,  the  Engineering  Development  Model, 
and  Production  Representative  Models. 


Figure  2.  Notional  System  Performance  Growth  Over  Developmental  Timeline 

Alternatively,  test  milestones  could  also  be  used.  While  the  SoS  PM  may  not  have 
specific  insight  into  present  performance  status,  at  a  minimum  performance  should  be 
expected  to  match  those  required  of  the  system’s  KPP  thresholds  when  a  system  enters 
production.  The  individual  system  capabilities  can  then  scaled  by  the  SoS  PM  in  terms  of 
their  knowledge  of  the  stage  of  the  individual  system’s  maturity  in  development,  any  data 
relevant  to  its  performance  at  that  stage,  and  how  that  performance  is  expected  to  be 
enhanced  or  degraded  by  incorporation  into  the  SoS.  This  adjudication  of  systems  is 
represented  through  the  use  of  a  weighting  function  to  represent  the  individual  capabilities’ 
maturity  (real  or  anticipated)  for  each  level  of  development  and  could  be  expressed  as 
follows: 


Pm  =  u>„*  a  (3) 

where  P=system  performance  in  the  SoS  as  a  function  of  the  adjustment  weight  (con)  and 
the  systems  expected  performance  (a)  at  production.  Independent  of  how  the  weights  are 
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assigned,  the  range  of  the  weighting  factors  (u)J  will  need  to  be  set  within  a  scale  from  0  to 
1 .0  so  that  when  full  capability  is  provided,  Pi  =  a. 

While  this  method  works  well  for  systems  that  are  not  integrated  with  others  to 
deliver  required  capability  in  an  SoS,  this  issue  becomes  more  complex  when  two  or  more 
systems  must  come  together  to  provide  a  level  of  capability  for  the  SoS.  This  is  due  in  part 
to  the  fact  that  the  integration  of  the  capabilities  into  a  single  capability  can  result  in  the 
combined  capabilities’  performance  being  either  enhanced  or  degraded.  Where  systems 
become  integrated  into  a  single  capability  that  provides  the  performance  value  it  can 
expressed  as: 

Pm(x,y,..)n—  COn  G(x,y,...)  (4) 

where  Pm(Xiyi..)„  represents  the  level  of  capability  that  is  comprised  of  systems  (x,y,  and  ..) 
contributing  towards  the  satisfaction  of  the  stated  performance  requirement  given  the  level 
of  maturity  of  the  system  or  SoS  (n)  and  the  anticipated  maximum  level  of  performance 
expected  from  that  combined  capability.  The  value  a(Xjy ...)  now  represents  the  maximum 
level  of  performance  capability  the  combined  capability  is  expected  to  eventually  satisfy. 

Operational  Employment  Components  (Usage  Options  and  Usage  Rate).  The 

operational  employment  function  of  the  SPM  is  to  define  and  address  the  impact  of  various 
CONOPS  on  the  SoS  and  its  ability  to  satisfy  the  KPP  requirements.  Because  one  of  the 
strengths  of  an  SoS  is  its  inherent  flexibility,  in  that  component  systems  can  be  organized  to 
solve  the  capability  problem,  this  can  often  be  translated  to  where  the  individual  capabilities 
may  be  used  in  varying  ways  to  accomplish  the  same  mission.  These  varying  CONOPS 
result  in  increased  complexity  for  the  analyst  in  trying  to  predict  what  level  of  performance  is 
being  achieved.  To  address  this  issue  the  SoS  PM  should  seek  operator  input  in  the 
development  of  a  set  of  scenarios  that  represent  the  range  of  potential  operational  usage 
concepts  for  the  individual  systems  within  the  SoS  as  applicable  to  each  KPP  and  increment 
of  development.  This  enables  the  derivation  of  a  set  of  equations  relating  the  KPPs  to  their 
component  systems  and  to  a  specific  CONOP.  The  set  of  CONOPS,  each  reflecting  an 
anticipated  level  of  performance  and  technical  maturity/integration  of  a  specific  capability  (x) 
at  a  specific  point  in  time  (n)  can  then  be  matrixed  together  to  enable  a  calculation  of  the 
overall  predicated  performance  of  the  SoS  across  a  range  of  scenarios.  For  example,  for  a 
specific  performance  metric,  several  CONOPS  reflecting  various  usage  options  and  rates 
may  be  developed  and  expressed  as  follows: 

CONOPS  Xn  =  PP In  +  qP2 n  +  5P3n  +  £P4n  +  yPsn  (5) 

where  CONOPS  X  represents  SoS  maturity  level  n,  Pxn  represents  the  anticipated  level  of 
performance  of  a  specific  capability  (x)  at  a  specific  point  (for  this  example  as  represented 
by  a  specific  SoS  state)  in  time  (n).  The  symbols  y,  5,  e,  p,  and  q  represent  pre-defined  and 
documented  usage  values  of  the  system  (integrated  or  standalone  as  appropriate)  for  each 
operational  scenario/performance  factor  derived.  If  the  notional  system  is  not  used  in  a 
specific  CONOP  then  the  associated  y,  5,  e,  p,  or  q  value  equals  zero. 

A  Notional  Case  Analysis  of  SPM  for  an  Anti-Submarine  Warfare  (ASW)  SoS 

Having  defined  the  contributing  factors  towards  determining  SPM  and  its  potential 
use  for  an  SoS  PM,  let  us  now  explore  a  notional  example  of  how  it  may  be  applied.  For  this 
analysis  we  look  at  a  subset  of  a  notional  ASW  SoS  based  loosely  on  the  capabilities  that 
were  being  developed  by  PMS  420;  all  performance  values  and  weightings  are  created 
purely  for  this  academic  exercise.  Based  on  the  elements  contributing  to  the  definition  of 
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what  constitutes  the  components  of  an  SPM  value,  we  can  break  it  down  into  10  process 
steps  for  determining  the  value  for  the  SoS  being  analyzed.  These  steps  are  as  follows: 

1 .  Define  the  notional  SoS  composed  of  “n”  systems  (what  systems  are  in  the 
SoS?); 

2.  Develop  the  notional  mission  strings  (defines  the  system  level  elements  of 
the  SoS  that  contribute  towards  the  individual  SoS  missions/functions/mission 
strings); 

3.  Map  system  level  contributions  towards  desired  SoS  performance  attributes 
(answering  the  question  of  how  do  the  individual  system  capabilities 
contribute  to  satisfying  the  SoS  requirements); 

4.  Define  the  notional  system  maturity  growth  paths  in  terms  of  an  expected 
developmental  capability/  performance  (reflecting  typical  product  growth 
paths  and  the  impact  of  the  integration  of  systems  to  provide  a  capability 
component); 

5.  Account  for  where  individual  systems/technologies  must  be  integrated  to 
support  the  functional  thread; 

6.  Develop  a  performance  corollary  to  reflect  where  multiple  technologies  work 
together  to  provide  a  unified  capability; 

7.  Define  the  methodology  for  mapping  the  performance  factors  and  their 
associated  technologies  to  potential  CONOPS  (usage  options/rates); 

8.  Combine  and  normalize  the  outcomes  from  the  CONOPS  analysis  to  provide 
a  single  point  metric  indicating  the  performance  expectation  of  the  defined 
SoS  state; 

9.  Use  the  predicted  system  maturation  paths  anticipated  in  the  SoS  to  predict 
the  probability  that  the  production  SoS  will  be  able  to  satisfy  its  performance 
metrics;  and  then 

10.  Combine  and  normalize  the  calculated  values  to  arrive  at  a  single  point 
prediction  on  whether  the  SoS  can  provide  the  required  performance  related 
to  the  specified  KPP. 

For  this  notional  case,  consider  an  ASW  SoS  (Step  1)  comprised  of  n=5  components 
systems,  as  indicated  in  Table  1 ,  which  combine  together  to  enable  the  conduct  of  the 
traditional  detect  to  engage  sequence  of  ASW  operations.  As  shown  in  Table  1,  Steps  2  and 
3  have  been  completed  by  mapping  the  individual  systems  to  where  they  are  expected  to 
contribute  to  fulfilling  the  various  KPPs  identified  (search,  detect,  classify,  and  engage).  For 
simplicity,  only  the  search  KPP  will  be  evaluated,  which  has  an  arbitrarily  assigned  desired 
threshold  of  400  square  miles  per  hour.  For  this  case,  only  four  of  the  systems  have 
application  to  the  search  KPP.  The  four  systems  are  comprised  of  three  sensors  (a  passive 
towed  array  [System  1],  a  vehicle  mounted  dipping  sonar  [System  2],  and  an  airborne 
dipping  sonar  [System  3])  that  can  contribute  towards  fulfilling  the  search  KPP.  The  passive 
array  represents  a  developmental  technology,  and  the  vehicle  mounted  dipping  sonar 
represents  a  mature  capability  that  is  repackaged  for  this  SoS  and  the  airborne  dipper 
reflects  a  system  in  production. 

Table  1 .  Notional  ASW  SoS 
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KPP  Impacted 

Capability/MS 

Search 

Detect 

Classify 

-ngage 

System  1 

(USV-Towed 

Array) 

X 

X 

System  2 
(USV- Dipper) 

X 

X 

X 

System  3 
(eOR-Dipper) 

X 

X 

X 

X 

System  4 

X 

System  5 
(USV) 

X 

X 

X 

With  respect  to  Step  4,  the  maturation  path  is  defined  to  reflect  the  growth  in 
capabilities  expected  to  where  at  production  capability  is  normalized  to  a  1 .0.  Table  2  maps 
this  and  is  used  to  present  a  notionally  assigned  weighting  (co)  value  of  the  individual 
capabilities. 


Table  2.  Notional  ASW  SoS  Maturity  Growth  Plan 


□eve lop  menial  status  (n  state) 

Capability  (oj) 

Present 

{0=1 ) 

Test 

Event  1 

tn=2) 

OPEVAL 

<n=3) 

Production 

(n=4) 

System  1 

0.7 

0.8 

0.9 

10 

System  2 

o.e 

0.9 

1.0 

1.0 

System  3 

1.0 

1.0 

1.0 

1.0 

System  5 

0.5 

0.7 

0.9 

1.0 

For  Step  5,  note  that  for  the  first  two  systems  (Systems  1  and  2),  an  integration 
challenge  exists  because  they  are  to  operate  off  of  an  Unmanned  Surface  Vehicle  which  is 
developmental  (System  5).  For  Step  6,  there  are  two  system  combinations  (Systems  1  and 
5  [USV/TA]  and  Systems  2  and  5  [USV/Dipper])  that  have  to  be  considered  and  one  system 
that  operates  as  a  standalone  capability.  For  simplicity  in  this  example,  in  integration  of 
capabilities,  the  lower  weighting  (oo)  value  of  the  individual  capabilities  has  been  assumed  to 
be  the  driving  factor  and  thus  the  value  for  the  combined  capabilities.  For  a  more 
comprehensive/complete  analysis,  the  SRL  methodology,  or  other  methodology,  maybe 
used  to  develop  this  insight.  For  the  performance  value  (a(X,y,...)),  sensor  performance  is 
taken  as  the  primary  parameter  of  interest  and  (for  this  example)  it  represents  the  maximum 
level  of  performance  capability  as  based  on  the  maximum  capability  that  the  sensor  (TA  or 
Dipper)  is  expected  to  eventually  satisfy.  Then,  for  performance,  using  the  equations  from 
the  Accounting  for  System  Performance  section  means  that  the  expressions  become: 

USV/TA  =  Pm(i,5)„=  u)„ *  a(Xjy,...)  =  Pm(1,5)n=  aj5n *  a(1)  (6) 

USV/Dipper=  Pm(2,5.)„=  *  a(x,y,...)  =  Pm(2,5)n=  w5n  *  a(2)  (7) 

MH60R  Dipper=  P3  =  Wn*  a  =  cjo3n*  a3  (8) 

Again  to  simplify  the  example,  it  is  assumed  that  supportability  is  not  impacted  by  the 
combination  of  the  systems  into  the  SoS.  Therefore,  there  is  no  need  to  adjust  the  weights 
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(co)  to  account  for  such  an  impact.  The  next  Step  (7)  now  requires  adjudication  on  how 
these  systems  will  be  potentially  used  in  real  world  operations.  Table  3  provides  a  notional 
mapping  to  address  the  impact  of  potential  operational  usage  concepts  (or  CONOPS)  on  the 
functional  capabilities  for  the  search  KPP. 


Table  3.  Mapping  CONOPS  to  Capabilities  Usage 


KPP.'Capability  -Search 

CONOR 

A 

CONOR 

B 

CONOR 

C 

Integrated  System  1  -  USV 
with  Towed  Array 

100% 

50% 

50% 

Integrated  System  2-  USV 
with  Dipping  Sonar 

50% 

25% 

System  3 -MH-60R  with 
Dipping  Sonar 

25% 

This  data  leads  to  the  equations  for  the  three  defined  CONOPS  as  follows: 

CONOPSa=  1.0(  u>5n*a(1));  (9) 

CONOPSb=  0.5(u)5n *  O(i))  +  0.5(oo5n  *  0(2)) ;  and  (1 0) 

CONOPSb=  0.5((jo5„ *  a(1))  +  0.25((jo5„ *  a(2))  +  0.25(co3n *  a3)  (1 1 ) 

These  equations  could  then  be  brought  together  to  provide  a  point  indication  of  the 
performance  to  be  expected  across  the  range  of  CONOPS  and  systems  used  for  executing 
that  specific  KPP  as  follows  for  the  search  example: 


SPMsearch„  =  {  CONOPSA(n)  ,  CONOPSB|n),  CONOPSc(n)  }  = 

0.5  ujV| 

0 

O.Suj5j] 

a  o 

■ 

JtQ 

0.5  lOS(] 

0.25  U)5„ 

0.25 

«3 

(12) 

Now,  assuming  that  the  predicted  performance  of  each  of  the  systems  as  a(1 )  =  600 
nm2/hr,  a(2)  =100  nm2/hr,  and  a3  =  300  nm2/hr,  it  is  straightforward  to  complete  the 
calculation  and  arrive  at  single  point  values  across  the  SoSs  pre-defined  evaluation  spots 
(n=  1 , 4  in  Table  2)  to  see  if  the  SoS  is  on  track  to  develop  the  desired  performance 
capability.  Table  4  provides  the  notional  results  from  the  above  analysis.  This  data  can  then 
be  plotted,  and  a  curve  representative  of  the  traditional  TPM  curve  can  be  derived,  as 
shown  in  Figure  3.  Although  not  an  exact  indication  of  the  performance  that  will  be  achieved 
by  the  SoS,  the  PM  now  has  insight  into  the  probability  of  whether  the  system  will  meet  the 
KPP  and  at  what  stage  of  development  under  various  operator  use  scenarios.  For  the 
example  used,  the  PM  could  now  have  reasonable  certainty  that  the  SoS  should  be  able  to 
satisfy  the  designated  KPP  by  production  and  could  provide  recommendations  to  the 
operator  on  more  effective  operational  modes  for  enhancing  the  SoS’s  performance. 

Table  4.  Notional  ASW  SoS  PML  Over  Time 
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Table  4:  Notional  ASW  SoS  PML  over  time 

Capability 

(nnChr) 

Present 

(n=1) 

Test 
Event  1 
<n=2) 

OPEVAL 

(n=3) 

Production 

<n=4) 

CONOPS  A 

300 

420 

540 

600 

CONOPS  B 

175 

245 

315 

350 

CONOPS  C 

200 

280 

360 

400 

Normalized 

Average 

225 

315 

405 

450 

Figure  3.  SPM  of  the  Search  KPP  as  a  Function  of  CONOPS  and  SoS  Maturity 
Conclusion 

SoS  use  continues  to  expand  across  the  DoD,  providing  significant  advantages  in 
increasing  the  acquisition  community’s  ability  to  provide  enhanced  capability  with  reduced 
cost  and  developmental  time.  However,  because  most  DoD  SOS  acquisitions  are 
acknowledged  SoSs,  some  significant  management  challenges  exist  for  the  SoS  PM. 
Whereas  tools  are  being  developed  to  support  the  PM  in  management  of  their  efforts,  one 
area  where  a  tool  appears  to  be  missing  is  in  assisting  the  PM  in  understanding  where  they 
stand  with  respect  to  being  able  to  provide  the  end  capability  desired  by  the  SoS  as 
expressed  by  the  KPPs.  As  metrics  have  been  shown  to  be  beneficial  tools  for  PMs  in 
managing  their  challenges,  a  metric  for  predicting  performance  called  the  SoS  Performance 
Measure  is  proposed  that  leverages  knowledge  the  SoS  PM  should  be  able  to  obtain. 
Although  significant  up-front  evaluation  will  be  required  by  the  SoS  program  office  to 
implement  the  SPM  methodology,  the  investment  of  time  helps  to  establish  a  common 
baseline  for  monitoring.  As  a  tool  (metric),  SPM  appears  to  provide  the  PMs  with  insight  into 
their  SoS,  enabling  more  effective  management  of  risk  and  more  effective  prediction  of 
when  performance  will  be  achieved.  Additionally,  as  a  tool,  SPM  could  potentially  be  used  to 
model  various  operational  and  technical  options  to  aid  in  the  understanding  of  the  potential 
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impact  of  the  proposed  changes.  A  graphic  similar  to  the  traditional  TPM  graphic  can  then 
be  constructed  by  calculating  composite  KPP  values  for  each  MP  increment  and  plotting  the 
composite  level  of  performance  against  time.  It  should  be  remembered  that  the  intent  behind 
SPM  is  to  provide  the  SoS  PM  with  the  necessary  insight  into  developmental  performance 
compared  with  documented  performance  requirements.  As  with  all  predictive  models,  the 
analysis  will  need  to  be  compared  against  measured  test  data  as  it  becomes  available  in 
order  to  verify  predictions  and  identify  whether  the  program  is  on  course  to  meeting  its 
stated  performance  requirements.  A  projected  further  expansion  of  this  methodology  will  be 
to  allow  for  the  evaluation  of  new  capabilities  prior  to  their  being  incorporated  into  a  planned 
upgrade  or  replacement  of  an  existing  capability. 
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System  of  Systems  Definition 


•  An  SoS  is  defined  as  a  set  or 
arrangement  of  systems  that 
results  when  independent  and 
useful  systems  are  integrated 
into  a  larger  system  that 
delivers  unique  capabilities 
[DoD,  2004(1)]. 
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ltas(gra0LiIlLC  St  hi  ml 
:nry-  CA 


SoS  Acquisition  Challenges 

•  SoS  acquisition  management  -  a  significant  increase  in 
complexity  over  traditional  system  acquisition 

•  Development  requires  that  significant  numbers  of 
technologies  be  integrated  tu  ui  ai  iumci 

•  Challenges  traditional  development  monitoring  tools  and  cost 
models 

-  need  to  capture  integration  complexity 

-  level  of  effort  required  to  connect  individual  components 

•  Unintended  Consequences  -  high  degree  of  inter-linkage 
between  components  can  cause  unintended  impacts  to 
overall  system  performance 

-  components  are  modified  from  original  use 

-  Technology  change:  replaced  throughout  the  system  life  cycle 


The  result  of  this  acquisition  management  paradigm  shift  has 
been  si  ni  icant  schedule  and  cost  overruns  in  SoS  ro  rams 
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What’s  Done  today- 

Technical  &  Financial 

Various  tools  and  metrics  are  used  to  monitor  the  status  of 
system  level  development/risk: 

•  Technology  Readiness  Level 

•  Earned  Value  Management 

•  Manufacturing  Readiness  Level 

•  Systems  Readiness  Levels 

•  integration  Keacuness  Levels 


System  Test,  Launch 
&  Operations 


System/Subsystem 

Development 


Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


Basic  Technology 
Research 


What’s  Done  today  -  Performance 


Mission  Needs/Critical 
Operating  Issues 

Key  System 
Performance 
Parameters 

Measures  of  Performance 
(MOP’S) 

x1 

Technical  Performance 
Measures  (TPM’s) 

PSM/INCOSE  Technical  Report.  (2005).  Technical  Measurement.  RoedlerG.J.  and 
Jones,  C. 


TPMs  track  the  key  indicators  of  system 
performance  versus  planned  progress  of  Key 
Performance  Parameters  and  other  key 
effectiveness  measures 


U.S.  Department  of  Defense  .  (2003).  Extension  to:  A  Guide  to  the  Project 
Management  Body  of  Knowledge.  Ft.  Belvoir,  VA:  Defense  Acquisition 
University  Press 


TPMs:  Used  to  Provide  PM  insight  into  likelihood  oi  achieving  nesirea 

Performance  (a  metric) 
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What  is  a  Metric  from  the  PM  Viewpoint 


Definition  of  METRIC 


1  plural :  a  part  of  pjosodv  that 


jtrical  _tructure 


T a  standard  of  measurement  <no  metric  exists  that  can  be  applied 
to  happiness  —  Scientific  Monthly> 


3:  a  mathematical  function  that  associates  a  real  nonnegative  number 
analogous  to  distance  with  each  pair  of  elements  in  a  set  such  that  the 
number  is  zero  only  if  the  two  elements  are  identical,  the  number  is  the 
same  regardless  of  the  order  in  which  the  two  elements  are  taken,  and 
the  number  associated  with  one  pair  of  elements  plus  tndi  dooudated 
with  one  member  of  the  pair  and  a  third  element  is  equal  to  or  greater 
than  the  number  associated  with  the  other  member  of  the  pair  and  the 
third  element 

Synonyms:  bar,  barometer,  benchmark,  criterion,  gold  standard,  grade, 
mark,  measure,  standard,  par,  touchstone,  yardstick 


Often  Program  Specific  &  more  useful  to  gain  insight  into  trends 
than  in  use  as  specific  values  data  points 


for  Informed  Cliimgc 


Naval  Liiiic  ScEmol 

Montcrcv. 


Need  -  SoS  PM  ability  to  predict  performance 


How  can  I  rapidly  and  reliably  gain 
insight  into  if  my  program  is  on  track  to  meet  my 
performance  requirements? 


Performance  Prediction  is  an  issue  during  SoS  Development 


Occasionally  done  by  Modeling  &  Simulation  (M&S)  of  the  proposed  design 

•  For  High  Fidelity  Results  is  extremely  costly 

•  Limited  ability  t„  verify  models  until  product  development  is  complete 

Somewhat  monitored  through  the  use  reported  system  level  Technical  Performance 
Measures  (TPMs). 

•  SoS  erformance  is  however  not  necessaril  e  ual  to  the  sum  of  s  stem  level 

J_  A  m/ 

performance 

•  Not  all  data  is  necessarily  provided  to  the  SoS  Program  Manager  (PM), 
especially  in  acknowledged  SoS’s  where  the  PM  does  not  have  direct 
control/authority  over  the  System  Level  PM’s. 


Acquis  i[it  >11  Ecscarch  Program:  Creating  Synergy  for  I  informed  Change 


Natill  h^i^TJllLtaiC  SclkiKrl 
Monterey.  Q 


Do  existing  metrics  answer  this  need? 


I  asked,  How  can  I  tell  if  my  program  is  on 
track  to  meet  my  performance  requirements? 


No-TRL  s,  oivl  s,  i  rivi  s,  etc  at  the  system  level 
provide  insight  into  potential  of  achieving  system 
level  performance  but ; 

•  not  the  impact  of  their  existing  performance 
level  or, 

•  how  performance  is  impacted  when  system 
capabilities  are  combined  into  a  SoS  or, 

•  understanding  of  how  various  combinations 
and  usage  rates  of  the  components  systems 
may  impact  the  overall  performance  results 


AoquMion  Ecstarch  Program:  Creating  Synergy  for  Informed  C  hange 


Naval  rs^iKTJilLLaic  School 
Montcrejp  CA 


So  how  can  we  solve  the  problem  of  providing 
the  PM  with  Insight? 


Proposed  Methodology 

1 .  Identif  the  ke^  factors  related  to  SoS  Performance 

2.  Develop  a  non-linear  formulation  that  will  support  the 
prediction  of  a  notional  SoS’s  Performance  over  time  under 
various  operational  usage  concepts  and  technology  mixes. 

3.  Identify  and  document  the  constraints  on  the  non-linear  model 
that  would  be  required  for  a  linear  approximation  model  to  be 
valid. 
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So  how  can  we  look  at  the  factors  determining 
SoS  Performance? 


Lets  assume  that  SoS  Performance  can  be  defined  as: 
f(  SoS  capability,  operational  employment) 

Where: 

SoS  capability  =  f(SoS  technical  maturity,  SoS  integration,  SoS 
support,  &  System  Ferformance)  where  the  individual  systems 
contribution/impact  to  the  SoS  can  be  determined  and, 

Operational  Employment  =  / (usage  options  (can  a  system  in  the 
SoS  help  meet  a  performance  goal),  usage  rate  (how  much  will  it 
be  used)) 


A  Ten  Step  Plan  for  predictina  SoS  Performance 


1)  Define  the  notional  SoS 
composed  of  “n”  systems 

2)  Develop  the  notional  mission 
strings 

3)  Map  system  level  contributions 
towards  the  desired  SoS 
performance 

4)  Define  the  notional  system 
maturity  growth  paths  in  terms  of 
a  expected  developmental 
capability/  performance 


Notional  System  of  Systems 

Key  Performance  Factor  Impacted 

Capability/ 

System 

Factor 

1 

Factor 

2 

Facto 

r  3 

Facto 

r  n 

Technology  1 

X 

X 

X 

Technology  2 

X 

X 

Technology  3 

X 

X 

X 

X 

Technology  4 

X 

Technology  5 

X 

X 

5)  Account  for  where  individual 
systems/technologies  must  be 
integrated  to  support  the 
functional  thread 

6)  Develop  a  performance  corollary 
to  reflect  where  multiple 
technologies  work  together  to 

,  rovide  a  unified  ca,  ability 


p  =  (V)  *  n 

rl  n  %  U 


Pm 


=  co„  *  a 


(x,y  ,..)n  n  (x,y,...) 
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A  Ten  Step  Plan  for  predicting  SoS  Performance 


7)  Define  the  methodology  for  mapping  the 

performance  factors  and  their  CONOPS  Xn  =  pPlw  +  r\P2n  +  5P3h  +  eP4w  +  yP5w 

associated  technologies  to  potential 

CONOPS 


8)  Combine  and  normalize  the  outcomes 
from  the  CONOPS  analysis  to  provide  a 
single  point  metric  indicating  the 
performance  expectation  of  the  defined 
SoS  state 

9)  Use  the  predicted  system  maturation 
paths  and  their  anticipated  insertion 
points  into  the  SoS  to  predict  the 
probability  that  the  production  SoS  will 
be  able  to  satisfy  its  performance 
metrics 

10)  Combine  and  normalize  the  calculated 
values  to  arrive  at  a  single  point 
prediction  on  can  the  SoS  provide  the 
required  performance  related  to  the 
specified  KPP 


SPM 


search 


=  (C0N0PS#|ll,  CONOPS,  (n),C0N0PSc|ll}  = 


0 

0 

— 

aID 

0.5  w5f] 

0.5o)Sn 

0 

X' 

“pi 

0.5  w5n 

0*25  0)5fJ 

0.25 

a3 

Performance  Factor  “n” 

=  [CONOPa,  CONOPb,CONOPc] 
=AVG(CONOPa+CONOPb+CONOPc) 
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Beginning  tool  for  gaining  insight  on  SoS 
Performance  for  use  in  Prediction  &  Monitoring 


Now  I  can  see  my  program  is 
potentially  on  track  to  meet 
my  performance  requirements 
but  may  have  risk 


Performance  Factor  “n” 
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Sample  Case  -  ASW  Search  Mission  for  a 
Notional  ASW  SoS 


N=5  SoS 


Table  1:  Notional  ASW  SoS 

KPP  Impacted 

Ga  pability/MS 

•Search 

De:ec: 

Classify 

Engage 

Syste  m  1 

(USV-Towed 

Array) 

X 

X 

Syste  m  2 
(U  5  V- Dipper) 

X 

X 

X 

Syste  m  3 
(60  R- Dipper) 

X 

X 

X 

X 

Syste  m  4 

X 

Syste  m  5 
(USV) 

X 

X 

X 

Search  functional  tread 
identifies  N=4  technologies 
of  which  two  sets  are 
integrated 


Maturation  pathways  are  developed 


Use  of  technologies  within  various 
CONOPS  determined 


Table  3:  Mapping  CONORS  to  Capabilities  Usage 


KPP.-Capability  -Search 

CONOP 

A 

CONOP 

e 

CONOP 

c 

Integrated  System  1  -  USV 
with  Towed  Array 

100% 

50% 

50% 

Integrated  System  2-  USV 
with  Dipping  Sonar 

50% 

25% 

System  3  -MH-60IR  with 

Dipping  Sonar 

25% 

1 1  tiJ  1 W 

.1  Wf 

^t±1£1!U 

Table  2:  Notional  ASW  SoS  Maturity  Growth 
Plan 

Developmental  status  (n  state) 

Capability  ((d) 

Prese  nt 

(n=1) 

Test 

Event  1 

f  n = 2 : 

OPEVAL 

(n=3) 

Production 

<n=4) 

System  1 

0.7 

0.8 

0.9 

1.0 

System  2 

0.8 

0.9 

1.0 

1.0 

System  3 

1.0 

1.0 

1.0 

1.0 

System  5 

0.5 

0.7 

0.9 

1.0 

Performance  Equations  developed 


USV/TA  =  Pm(lj5)w=  <»w  * 


USV/Dipper=  Pm(2  5> 

MH60R  Dipper=  P3 


a(x,y,...)=Pm(l,5)«= 


00 


*  ®(x,y,...)  =  Pm(2,5)n= 


a, 


5  n 


(i) 

a, 


ooH*a 


(2) 


CO 


3  n 


ou 


CONOPS  Equations  developed 


CONOPSA=  1.0(  co, 


CONOPSB=  0.5(co 


J5  n 


a 


CONOPSb=  0.5(co 


(i) 


J5  n 


a(1))  +  0.25(co 


)  +  0.5(co 

5  n 


* 


(iy 

5  n 

a(2))  +  0.25(g)3m  *  a3) 


a(2)) ;  and 
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Sample  Case  -  ASW  Search  Mission  for  a 
Notional  ASW  SoS 

Now  assuming  the  predicted  performance  in  production  of  each  of  the  systems  was 
a(l)  =  600  nm2/hr,  a(2)  =100  nm2/hr,  and  a3  =  300  nm2/hr 


r-  - - 

0 

0 

“(1} 

SPMsearch„  =  {  CONOPSA(n,,  CONOPSw  COMOPSc{n)  }  = 

0.5  (05r] 

0.5  U)5n 

0 

x- 

“(Zf 

0.5  (U5n 

0.25  OJ5n 

0.25  Ulqjj 

“3 

Table  4:  Notional  ASW  SoS  PML  over  time 

Capability 

(nm2ir) 

Present 

(n=l) 

Test 
Event  1 
<n=2) 

OPEVAL 

<n=3) 

Production 

tn=4) 

CONOPS  A 

300 

420 

540 

600 

CONOPS  B 

175 

245 

315 

350 

CONOPS  C 

200 

280 

360 

400 

Normalized 

Average 

225 

315 

405 

450 

-♦-CONORS A  -B-CONOPSB  -±-C0N0PSC  -*-AVGPMl 

700  - 


12  3  4 

Figure  3:  SPM  of  the  Search  KPP  as  a  Function  of 
CONORS  and  SoS  Maturity 
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A  word  of  Caution:  Any  Metric  is  Fallible 


TRL  6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 


System  Test,  Launch 
&  Operations 


System/Subsystem 
Develop  me  nr 


Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


Basic  Technology 
Research 


But  a  fuller  definition  is:  Representative  model  or  prototype 
system,  which  is  well  beyond  the  breadboard  tested  for  TRL  5,  is 
tested  in  a  relevant  environment.  Represents  a  major  step  up  in  a 
technology's  demonstrated  readiness.  Examples  include  testing  a 
prototype  in  a  high  fidelity  laboratory  environment  or  in  simulated 
operational  environment. 


A  relevant  environment  is  a  set  of  stressing  conditions,  representative  of 
the  full  spectrum  of  intended  operational  employments,  which  are  applied 
to  a  GTE  as  part  of  a  component  (TRL  5)  or  system/subsystem  (TRL  6) 
to  identify  whether  any  design  changes  to  support  the  required  (thresh¬ 
old)  functionality  are  needed. 


DEPARTMENT  OF  DEFENSE 


Technology  Readiness 
Assessment  (TRA)  Deskbook 


Prgpartd  by  lh» 

Director,  Reeearcti  Directorate  (0RD) 

Office  ot  the  Director  Oatemo  R .March  and  Engineering  (DOft&E) 


Supporting  Information 


Afunctional  form  of  a  system,  generally  reduced  in  scale, 
near  or  at  operational  specification.  Models  will  be  suffi- ' 

rientlu  harHoneH  tn  all™  Hemnnctratinn  fifths  tof-hnif-al  anri 


Results  from  laboratory  testing  of  a  proto¬ 
type  system  that  is  near  the  desired  con¬ 
figuration  in  terms  of  performance,  weight, 
and  volume.  How  did  the  test  environment 
differ  from  the  operational  environment? 
Who  performed  the  tests?  How  did  the 
test  compare  with  expectations?  What 
problems,  if  any,  were  encountered? 

What  are/were  the  plans,  options,  or 
actions  to  resolve  problems  before 


Metric  should  be  used  as  an  indicator  of  “is  further  research 

into  this  area  needed ?” 
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Concluding  Thoughts 


•  System  of  Systems  (SoS)  management  is  an 

exce  tional  challen  e  for  toda  ’s  com  lex  &  inte  rated 
systems. 

>  SoS  use  however  is  increasing  across  DoD 

>  Metrics  ex  ioi  hi  1 1  icii  ty  af^ci  s  but  for  predicting  K<=.  .ormance, 
short  of  extensive  M&S,  are  still  in  development 

•  The  presented  SoS  Performance  Methodology  may 
offer  a  way  to  assist  the  SoS  Pivi  in  gaining  insignt  into 
this  area 


>  Understanding  of  SoS  architectures  and  how  technologies 
interact  is  key  element 

>  All  metrics  are  potentially  fallible  if  used  beyond  their 
limitations 
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Abstract 


Program  Managers  (PMs)  are  expected  to  quantifiably  justify  that  their  program  will  result  in  the  delivery  of  a  system  with 
the  required  performance  through  development. 

Traditionally,  the  PM  has  several  technical  management  tools  at  their  disposal,  including  TPMs,  Modeling  and  Simulation, 
etc.  that  provide  insight  and  predictive  capability  in  system  performance.  When  the  program  matures  to  a  point  where  actual 
test  data  can  be  gathered,  it  is  compared  against  expected  system  performance. 

The  increasing  use  of  the  System  of  Systems  (SoS)  model  for  the  rapid  fielding  of  warfighting  capabilities  poses  new  systems 
engineering  challenges  for  the  DoD.  Due  to  the  complex  nature  of  SoS  interdependencies,  PMs  are  especially  challenged 
when  asked  to  uantifiabl^  ^  redict  ^  ro^ress  made  toward  full-ca^  ability  SoS  ^  erformance  in  an  incremental  develo^  ment.  To 
support  the  PM  in  making  technical  trades  and  tracking  performance  progress  for  an  acknowledged  SoS,  the  US  Navy  (PMS 
420  and  SSC  Pacific)  have  been  collaborating  on  the  development  and  verification  of  a  SoS  Performance  Measure  (SPM) 
tool  set. 


The  SPM  tool  applies  a  modiiied  lecimical  Peiioniiance  Measure  (TPM)  type  approach  to  a  Soo  construct.  nowever,  instead 
of  focusing  on  a  single  measurable  technical  value  that  can  be  monitored  during  development  of  a  Individual  system,  the 
SPM  links  the  SoS  Key  Performance  Parameters  (KPPs)  to  individual  component  capabilities,  their  maturity,  and  their 
potential  usage  rates.  The  System  Maturity  Model  (SMM),  Concept  of  Operations  (CONOPS),  and  usage  rate  variance 
analyses  are  all  considered  in  the  SPM  calculation.  The  SPM  tool  will  be  reviewed  and  valuable  lessons  learned  to  date 
within  the  Mission  Modules  Program  will  be  discussed. 
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What  is  being  developed  for  a  SoS 

what  metrics/  methods  presently  exist? 


STEVENS 


Institute  of  Technology 


TKtmotogy 


TRL5 
TRL  8 
TRL  7 


Has  Develo.  ed  a  Maturity  based  anal,  sis 

methodology  called:  System  Readiness 
Level  (SRL) 


SaKSSasa ssr 

Six" 

SRL  =  IRL  x  TRL 

(Normalized) 


Provides  a  system-level  view  of  development  maturity  with 
opportunities  to  drill  down  to  element-level  contributions 


Sauser,  B.,  J.  Ramirez-Marquez,  D.  Henry  and  D.  DiMarzio.  (2007).  “A  System  Maturity  Index  for  the  Systems 
Engineering  Life  Cycle.”  International  Journal  of  Industrial  and  Systems  Engineering.  3(6).  (forthcoming) 


Acquisition  Eestarth  Program;  Creating  Sy  nergy  for  Informed  Change 


\.LI'lLI  i’EtalRf-Jll  LLMC  St  Bn  h  hi 


